main_talk.knit

class: left, middle, inverse, hide-logo

### Boosting methods for regression

#### Emanuel Sommer
#### 2021-06-22

<center>
 
<img src="img/maTUM_w.png" width="100">
 
 
Department of Mathematics
 
Technical University of Munich
</center>

---
## Basic Notation

Training data set:

`$$\mathcal{D} = \{(y_i,x_i)\ | i \in [N]\}$$`

Response: `$y_i \in \mathbb{R}$` like the burnout rate or the charges.

Predictors: `$x_i \in \mathbb{R}^m$`

.pull-left[
.right[
`$\textbf{y} = \begin{pmatrix}y_1\\ \vdots \\ y_N \end{pmatrix}$`
]
]

.pull-right[
.left[
`$\textbf{x} = \begin{pmatrix}x_1\\ \vdots \\ x_N \end{pmatrix}$`
]
]

---
## Previously

.pull-left[
### CART

Let `$\mathcal{T}$` be the space of CART models.

`$$t(x_i, \gamma, R) = \sum_{j=1}^J \gamma_j I(x_i \in R_j) \quad  t \in \mathcal{T}$$`
with `$R_j$` being distinct regions of the predictor space and `$\gamma_j$` the prediction w.r.t. the region.
]

.pull-right[
### Random forest

Build many deep trees in parallel and average over the individual predictions. Randomness is induced to decorrelate the trees.

]
---
background-image: url("data:image/png;base64,#https://media.giphy.com/media/cRH5deQTgTMR2/giphy.gif")
background-size: cover
class: center, bottom, hide-logo

---

## The core idea of boosting

**Sequentially** build weak learners - here shallow trees - that are ultimately combined in an additive fashion. Each additional learner should improve the total model.

**The general additive model:**

`$$\hat{y}_i = \phi(x_i) = \sum_{k=1}^{K} t_k(x_i), \quad t_k \in \mathcal{T}$$`

---

## The fitting approach

To fit such a model one uses **Forward Stagewise Additive Modeling**.

* Given the current model `$\phi_{k-1}(\textbf{x})$`
* Add the tree `$t_k$` with regions `$R^{(k)}$` and leaf predictions `$\gamma^{(k)}$`
* The parameters of the current model are unchanged!

`$$\phi_k(\textbf{x}) = \phi_{k-1}(\textbf{x}) + t_k(\textbf{x}, \gamma^{(k)}, R^{(k)})$$`

Each additional tree should improve the overall model i.e. solve the following optimization problem:

`$$(\gamma^{(k)},R^{(k)}) = argmin_{\gamma,R} \sum_{i=1}^N L(y_i, \phi_{k-1}(x_i) + t(x_i,\gamma,R))$$`
where `$L$` is a convex and differentiable loss function.

---

## Connection with Gradient descent

**General minimization problem:**

`$$\textbf{s}_{opt} = argmin_\textbf{s} f(\textbf{s}) \quad \textbf{s} \in \mathbb{R}^N$$`

Where `$f$` is differentiable and `$\lambda \in \mathbb{R}$` a scalar.

**Gradient descent in a nutshell:**

$$
\textbf{s}^{(k)} = \textbf{s}^{(k-1)} - \lambda \bigg[\frac{\partial f(\textbf{s})}{\partial \textbf{s}}\bigg]_{\textbf{s}=\textbf{s}^{(k-1)}}
$$

For now set `$\lambda = 1$`.

---
## Connection with Gradient descent

.pull-left[
**General minimization setup:**

`$$\textbf{s}_{opt} = argmin_\textbf{s} f(\textbf{s}) \quad \textbf{s} \in \mathbb{R}^N$$`

Gradient descent:

$$
\textbf{s}^{(k)} = \textbf{s}^{(k-1)} - \bigg[\frac{\partial f(\textbf{s})}{\partial \textbf{s}}\bigg]_{\textbf{s}=\textbf{s}^{(k-1)}}
$$
]

.pull-right[
**The boosting setup:**

`$$\phi_{opt} = argmin_{\phi} \sum_{i=1}^N L(y_i, \phi(x_i))$$`

`$$\phi_k(\textbf{x}) = \phi_{k-1}(\textbf{x}) + t_k(\textbf{x}, \gamma^{(k)}, R^{(k)})$$`
]

Corresponding parts:

* The loss function `$L$` to the function `$f$`
* The predictions `$\phi_k(\textbf{x})$` to the input `$s^{(k)}$`
* The predictions of the tree `$t_k$` to the negative gradient of `$f$`

---

## Tree-based gradient boosting

**Algorithm:**

**1.**  Initialize `$\phi_0(\textbf{x})$` as a singular node tree.

**Example:**

Burnout data with the RMSE loss, fixed `$K = 2$` and tree depth 1.

---
## Tree-based gradient boosting

**2.**  For each tree addition `$k = 1$` to `$K$` do:

- For each observation `$i = 1$` to `$N$` compute:
  
    `$g^{(k)}_{i} = \bigg[\frac{\partial L(y_i, \phi(x_i))}{\partial\phi(x_i)}\bigg]_{\phi = \phi_{k-1}}$`

- Fit a regression tree by least squares to the outcome vector `$-\textbf{g}^{(k)}$` in order to get the `$J^{(k)}$` distinct regions `$\tilde{R}^{(k)}_j$`.

- For each of these `$J^{(k)}$` regions perform a line search in order to compute the leaf predictions `$\tilde{\gamma}^{(k)}_{j}$`.

- Set `$\phi_k(\textbf{x}) = \phi_{k-1}(\textbf{x}) + t(\textbf{x},\tilde {\gamma}^{(k)}_{j},\tilde{R}^{(k)}_j)$` with `$t \in \mathcal{T}$`

<!--
## Which loss to pick?

### `$L_2$` loss

➖ Sensitive to outliers.

➕ Simplifications possible as the gradient is the residuals.

.pull-left[
### Alternatives

* `$L_1$` loss
* Huber loss
]

.pull-right[
### Comparison

]
-->

---
## The example continued

---
## Combat Overfitting

### Early stopping

Monitor the loss on one or many (CV) validation sets and stop the addition of trees when the validation loss increases.

### Shrinkage

Introduce a learning rate `$\eta$` that scales the contribution of the new model.

`$$\phi_k(\textbf{x}) = \phi_{k-1}(\textbf{x}) + \eta * t(\textbf{x},\tilde {\gamma}^{(k)}_{j},\tilde{R}^{(k)}_j)$$`

### Subsampling

**Row-subsampling:** In each step use only a fraction of the training data.

**Column-subsampling:** In each step use only a fraction of the training features.

---

## An implementation: XGBoost

<center>
** A highly scalable end-to-end tree boosting system.**
</center>

* Strong regularization
  - Lasso/ Ridge penalization on `$\gamma$` 
  - Shrinkage ✔️
  - Subsampling ✔️
  - Early stopping ✔️
  - Maximum tree depth ✔️
* Parallelized tree building 
* Extremely efficient on sparse data and can handle missing values by having learned default directions in each split
* Cache aware algorithm

Also very popular in R: `gbm` package. (Same functionality: ✔️)

<!-- 
## XGBoost vs overfitting

### Regularized Loss

`$$\mathcal{L}(\phi) = L(\phi) + \sum_{k=1}^K \Omega_{\alpha, \lambda, \nu}(t_k)$$`

Where `$\Omega_{\alpha, \lambda, \nu}$` is the penalizing term for model complexity.

* `$\alpha$` is a `$L_1$` regularization parameter on `$\gamma$`.
* `$\lambda$` is a `$L_2$` regularization parameter on `$\gamma$`.
* `$\nu$` is the minimal loss reduction for terminal partitions.
* Second order approximations of the loss allow for better parallelization.

.pull-left[
### Shrinkage

### Subsampling
]

.pull-right[
### Early stopping

### Maximum tree depth
]
-->
---
background-image: url("data:image/png;base64,#https://media.giphy.com/media/BpGWitbFZflfSUYuZ9/giphy.gif")
background-size: cover
class: center, bottom, hide-logo

---
## Burnout data set

**Goal:** Predict the burnout rate of an employee.

.left-column[

**Number of predictors:**

**Training data:**

17301 rows

**Test data:**

4325 rows

**Missing:**

14%
]

.right-column[
<img src="data:image/png;base64,#C:/Users/Emanuel/Documents/Uni/3. MA Semester/Seminar/boosting_methods/_pictures/burn_rate_raw_nojitter.png" width="90%" style="display: block; margin: auto;" />
]
---

## Transformed outcome

The transformation with the empirical logit removes the bounds on the target.

`$$log(\frac{x+0.5}{1-x+0.5})$$`

<!--
## Mental fatigue score

Proposed baseline model:

$$
\text{burn_rate}_i = \beta_0 + \beta_1 \text{ mental_fatigue_score}_i + \epsilon_i \quad \text{with }
\epsilon_i \sim N(0,\sigma^2)
$$
-->

<!--
## Two closely related features

Both features show relevant main effects.
-->
---
## Prepare for takeoff

### Pre-processing / Feature Engineering

* All variables are used without normalization.
* Extract extra features `day of the week` and `month` of the date of joining.
* Dummy-encoding of all nominal variables.

### Resampling

**5-fold cross validation** is used for model evaluation e.g. hyperparameter tuning.

---
## Tuning strategy

**1.** `#Trees`

Fix all other hyperparameters to default values and the learning rate rather small to determine `$K \in [100,2000]$` to minimize the validation loss.

**2.** `Tree and the basic regularization hyperparameters`

.pull-left[
`Learning rate`: `$\eta \in [1e^{-10},0.1]$`

`Max tree depth`: `$max(J^{(k)}) \in [1,15]$`

`Loss reduction`: `$\nu \in [1e^{-10},30]$`
]

.pull-right[
`Column-subsampling`: `$[1,\text{#features}]$`

`Row-subsampling`:  `$[0.1,1]$`
]

Evaluate a set of combinations of these and determine the combinations that again minimize the validation loss. (Set of combinations: space-filling design)

**3.** Refine the set of combinations and repeat 2. to find the final combination.

---
## Tuning the number of trees

👉 Fix 1500 trees.

---
## Space-filling grid search

Tuning both the model with the raw outcome and the transformed one with space-filling grids came at a cost: **8 hours** of parallel computation.

**The final parameters:**

.pull-left[
`#Trees`: 1500

`Max tree depth`: 4

`Learning rate`: 0.02
]

.pull-right[
`Loss reduction`: roughly 0

`Column-subsampling`: not enabled

`Row-subsampling`:  80%
]

---
## Raw vs transformed target

The two models behaved exactly the same while tuning and show no notable differences in any of the following visualizations so the focus will now be on the raw model.

---
## Feature importance

The feature importance is basically the gain w.r.t. the loss of the feature over all splits.

---
## Handling of missing values

The default directions of XGBoost seem to work well.

---
## Final performance

---

## Insurance data set

**Goal:** Predict the medical costs (Charges) billed by health insurance.

.left-column[

**Number of predictors:**

**Training data:**

1071 rows

**Test data:**

267 rows

**Missing:**

None
]

.right-column[
<img src="data:image/png;base64,#C:/Users/Emanuel/Documents/Uni/3. MA Semester/Seminar/boosting_methods/_pictures/charges_ins_nojitter.png" width="90%" style="display: block; margin: auto;" />
]

---
## Fitting and tuning

- Again all variables are used without normalization.
- Dummy-encoding of all nominal variables.
- Resampling setup with 5-fold CV is the same.
- The tuning strategy and ranges are unchanged.
- The tuning w.r.t. the number of trees resulted in a value of 600.
- The space-filling grid search over the remaining 5 hyperparameters took 30 minutes in total.

**The final parameters:**

.pull-left[
`#Trees`: 600

`Max tree depth`: 3

`Learning rate`: 0.02
]

.pull-right[
`Loss reduction`: 0.03

`Column-subsampling`: 7 of 8

`Row-subsampling`:  80%
]

--

<!--
## Tuning the number of trees

So a closer look at the region around 500 is advisable.

## Tuning the number of trees

👉 Fix 600 trees.
-->
<!--
## A glimpse at learning rates

Thus a too small learning rate can also be a problem.
-->

<!--
## Space-filling grid search

- 600 combinations.
- 900k trees.
- 30 minutes of parallel computation.

**The final parameters:**

.pull-left[
`#Trees`: 600

`Max tree depth`: 3

`Learning rate`: 0.02
]

.pull-right[
`Loss reduction`: 0.03

`Column-subsampling`: 7 of 8

`Row-subsampling`:  80%
]
-->

<!--
`#Trees`: 600

`Max tree depth`: 3

`Learning rate`: 0.02

`Loss reduction`: 0.03

`Column-subsampling`: 7 of 8

`Row-subsampling`:  80%
-->

---
## Feature importance

---
## Final Performance

---
## Pros

- **Minimal pre-processing**

- **Flexible enough to detect complex non-linear patterns**

- **Handling of missing values**

- **Integrated feature selection**

- **Good generalization due to lots of regularization options**

- **Strong predictive power**

## Cons

- **Not easily explainable**

- **Computationally demanding (especially the hyperparameter tuning)**

- **Better if there are lots of observations**

---
class: inverse, center, middle, hide-logo

## Thank you and happy boosting!

Check out the bookdown project for more!

---
class: inverse, center, middle, hide-logo

## Discussion

---
## Interesting residuals

<div id="htmlwidget-8d63fd77d2292c277963" style="width:100%;height:504px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-8d63fd77d2292c277963">{"x":{"visdat":{"13a825b37ea1":["function () ","plotlyVisDat"]},"cur_data":"13a825b37ea1","attrs":{"13a825b37ea1":{"x":{},"z":{},"y":{},"text":{},"hoverinfo":"text","color":{},"symbol":{},"colors":["#CC4678FF","#0D0887FF"],"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"symbols":["x","circle"],"type":"scatter3d","mode":"markers","opacity":0.9,"size":2,"inherit":true}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"Age"},"yaxis":{"title":"BMI"},"zaxis":{"title":"Raw residuals"}},"title":"Raw test residuals of the insurance XGBoost model","legend":{"title":{"text":"Smoker"}},"hovermode":"closest","showlegend":true},"source":"A","config":{"showSendToCloud":false},"data":[{"x":[18,32,31,52,23,19,24,18,64,28,40,31,53,58,57,31,54,55,41,52,37,18,20,53,48,54,41,18,63,40,53,26,24,41,47,59,44,19,55,58,53,19,52,28,58,49,52,34,26,57,52,64,49,34,57,33,49,56,57,64,62,48,52,38,19,52,25,60,35,40,35,39,24,57,62,18,63,54,31,37,22,51,38,33,20,25,46,34,19,19,18,62,25,19,27,28,59,29,24,51,48,39,48,18,19,33,53,40,24,56,47,33,62,18,45,21,58,24,31,47,64,38,61,44,45,31,22,39,64,18,46,36,60,63,35,51,25,57,55,51,40,50,33,41,23,61,26,60,44,22,44,24,34,62,26,18,19,31,24,46,52,35,39,34,22,19,50,28,21,61,54,64,36,60,32,32,57,52,23,18,49,50,26,41,52,35,36,20,32,43,32,37,19,22,59,41,56,58,58,45,18,28,55,43,35,22,18,58,20,18,18,51,57,23,18],"z":[-559.895768193102,-285.798447994573,166.160130995318,-1640.34546472001,-841.651523903832,358.447628157622,-169.701965867938,-685.081443005199,16349.4056686668,-851.692829443274,-630.121902461979,649.245475345141,-452.425878910602,-880.912683262219,-910.731849348191,-674.572891311772,-163.325438080859,-2205.40390170798,36.2723567525627,-1208.78049151623,-941.930253981824,-113.909497029493,-191.837734497546,-753.384252870612,-2553.60361276063,-1250.74403447937,-123.883329375256,-595.89018115075,341.697428494565,-1218.13011500114,-1031.72258178435,-70.6149855363142,22468.5031571188,-373.123834813202,-1588.93863486007,-660.158989675843,-1458.67365176814,-3.91786930278226,-893.050836708253,-809.210589437957,7369.62281165481,-2606.77076780426,14041.3679410583,14764.7693435901,-597.996093318228,-1580.31807113277,-2056.82324085295,-1291.89613148762,18095.4773045501,-1719.98399330942,-1754.44106426169,1030.66515862553,-1048.59123360993,-427.982526652509,-797.531410734979,7295.34928621457,-1223.46757021579,-266.564344534811,-312.410136150553,1307.38303704728,13019.2141432638,-1422.45690857707,-2868.01579405431,81.2832558119817,-366.305118199377,-2138.8987348861,-274.861084923185,-1493.32988046182,-470.80782272924,-809.911007089193,-1322.42280813002,-1134.61649916576,-508.49528427246,-1134.90393368421,-2044.85137737716,-385.544020862301,-2618.68783885339,301.600056090625,-103.73665774984,-660.974884838872,-133.356362446797,-1285.87555137442,-571.525150974553,-289.278966909756,-1149.03383917436,14656.0284826427,-1073.90284508596,-1038.02324878726,-442.033568384102,-499.027511288761,-443.822468965725,16735.9909808734,67.3341905862767,-189.006796546968,-432.275690601863,-1557.52250871349,-838.544728485411,-252.249682422645,-737.826652866634,-914.959418759438,-884.52521405699,-699.718741028778,-1531.72950749062,-664.541092690685,-189.562796546968,-741.404083151854,-1349.60262615873,-693.386120146081,-387.804914584638,-826.896535287045,-1108.83551031424,-745.994750846805,-478.024898887792,-364.807492050718,-1309.16907944847,-241.059658766796,-516.088293490186,11587.8603010242,-938.896810272773,-1089.59547811426,516.90512356317,-211.530320187307,14257.0397157496,3875.86341226283,-1971.4171332944,-293.119337664407,186.178458810755,-812.039778157851,43.1967593313657,-725.978883562773,314.191336495951,-1020.05304791368,-1018.397174133,-1230.58334056061,-923.488425507269,-206.322216716055,14836.506636078,-1156.96702894438,-914.597768840076,-1266.22282907153,-632.500519165698,-1556.36669025607,-282.964617493845,-512.550192151025,-1347.71425127897,19.5674783858667,-488.456637490524,-532.143219107114,-1012.05218233062,-104.062123364966,-1212.32022903771,-120.250240677897,-1084.79113764454,-620.882156794403,136.521664733659,-1175.3865312323,-229.579594716033,-730.538419795329,9087.55011919952,-290.766328712811,13682.8364774795,-1367.11256404613,131.918320950323,-491.83622891452,172.58588479824,-161.314058819569,-665.737158187845,-1459.35424436821,-135.139800021246,-1745.89624219914,-52.2766259278251,528.317876464553,-318.739453299797,-206.737440092335,-715.755614900831,-511.282084148733,-925.349584204841,-961.382813943577,-776.757414875316,-982.541502990347,-1308.3291535601,-1120.1848119651,-271.895676021659,539.238964873791,15765.235406239,-978.920710768336,-95.3123198050644,-445.884874425795,-98.4091044249517,-983.708397936819,-486.430387347862,-662.698381734972,12483.2458632989,-100.634486318195,23496.4629312962,-579.585257359172,-1366.74477791316,-880.483090145364,136.916002460091,-869.632988496947,-631.362709349541,180.949020693814,16017.4165304662,-989.115460874464,-1239.17838050253,-28.6927859825091,-685.317753667974,-728.137358275964,-217.945051653008,-100.830546113186,8849.71320341624,-1148.4343694331,-1029.95396393204,7952.25535406233,-503.287510573287],"y":[33.77,28.88,25.74,30.78,23.845,28.6,26.6,35.625,24.7,25.935,36.19,28.5,28.1,32.01,34.01,26.885,30.8,38.28,31.6,32.205,23.37,23.75,28.975,35.9,29.7,39.6,32.2,34.43,31.8,41.23,26.6,29.92,23.21,31.635,25.46,27.83,38.06,20.9,25.365,25.2,38.06,20.615,26.4,27.5,34.865,35.86,33.25,25.27,29.64,40.945,36.7,34.5,41.47,29.26,23.18,35.245,30.78,32.3,22.23,30.115,31.46,31.445,38.38,21.12,17.48,34.1,30.59,33.11,28.9,29.6,38.6,29.6,23.4,30.495,38.095,24.09,41.325,21.47,23.6,30.875,31.35,22.42,28.025,38.9,31.79,41.325,33.44,34.21,35.53,30.495,37.29,36.86,23.465,20.7,25.175,26.98,28.785,26.03,29.3,39.7,30.78,26.22,31.13,33.66,20.3,18.5,26.41,41.69,23.655,33.725,29.545,32.9,37.4,40.28,24.035,28.975,22.77,33.63,27.645,32.3,39.7,19.475,36.1,36.48,39.805,29.26,23.18,31.92,35.97,30.14,30.8,34.43,24.32,33.1,23.465,34.2,32.23,28.1,33.535,25.4,29.9,36.2,33.44,28.8,27.36,44,27.265,35.1,32.34,28.31,27.5,33.99,35.815,39.16,46.53,26.18,22.61,39.49,26.79,24.795,36.765,27.1,34.32,23.56,20.235,40.5,31.6,29.26,34.6,38.38,23,26.41,28.595,18.335,27.835,31.5,31.54,47.74,32.7,31.35,29.925,26.22,30,32.6,24.86,35.815,22.135,30.59,41.1,34.58,35.2,34.105,27.93,32.11,34.8,23.94,34.43,30.305,23.3,27.83,33.33,24.3,37.715,29.9,27.61,30.4,30.03,25.175,22,26.125,28.31,30.03,25.74,33.4,36.85],"text":["Age: 18 BMI: 33.77","Age: 32 BMI: 28.88","Age: 31 BMI: 25.74","Age: 52 BMI: 30.78","Age: 23 BMI: 23.845","Age: 19 BMI: 28.6","Age: 24 BMI: 26.6","Age: 18 BMI: 35.625","Age: 64 BMI: 24.7","Age: 28 BMI: 25.935","Age: 40 BMI: 36.19","Age: 31 BMI: 28.5","Age: 53 BMI: 28.1","Age: 58 BMI: 32.01","Age: 57 BMI: 34.01","Age: 31 BMI: 26.885","Age: 54 BMI: 30.8","Age: 55 BMI: 38.28","Age: 41 BMI: 31.6","Age: 52 BMI: 32.205","Age: 37 BMI: 23.37","Age: 18 BMI: 23.75","Age: 20 BMI: 28.975","Age: 53 BMI: 35.9","Age: 48 BMI: 29.7","Age: 54 BMI: 39.6","Age: 41 BMI: 32.2","Age: 18 BMI: 34.43","Age: 63 BMI: 31.8","Age: 40 BMI: 41.23","Age: 53 BMI: 26.6","Age: 26 BMI: 29.92","Age: 24 BMI: 23.21","Age: 41 BMI: 31.635","Age: 47 BMI: 25.46","Age: 59 BMI: 27.83","Age: 44 BMI: 38.06","Age: 19 BMI: 20.9","Age: 55 BMI: 25.365","Age: 58 BMI: 25.2","Age: 53 BMI: 38.06","Age: 19 BMI: 20.615","Age: 52 BMI: 26.4","Age: 28 BMI: 27.5","Age: 58 BMI: 34.865","Age: 49 BMI: 35.86","Age: 52 BMI: 33.25","Age: 34 BMI: 25.27","Age: 26 BMI: 29.64","Age: 57 BMI: 40.945","Age: 52 BMI: 36.7","Age: 64 BMI: 34.5","Age: 49 BMI: 41.47","Age: 34 BMI: 29.26","Age: 57 BMI: 23.18","Age: 33 BMI: 35.245","Age: 49 BMI: 30.78","Age: 56 BMI: 32.3","Age: 57 BMI: 22.23","Age: 64 BMI: 30.115","Age: 62 BMI: 31.46","Age: 48 BMI: 31.445","Age: 52 BMI: 38.38","Age: 38 BMI: 21.12","Age: 19 BMI: 17.48","Age: 52 BMI: 34.1","Age: 25 BMI: 30.59","Age: 60 BMI: 33.11","Age: 35 BMI: 28.9","Age: 40 BMI: 29.6","Age: 35 BMI: 38.6","Age: 39 BMI: 29.6","Age: 24 BMI: 23.4","Age: 57 BMI: 30.495","Age: 62 BMI: 38.095","Age: 18 BMI: 24.09","Age: 63 BMI: 41.325","Age: 54 BMI: 21.47","Age: 31 BMI: 23.6","Age: 37 BMI: 30.875","Age: 22 BMI: 31.35","Age: 51 BMI: 22.42","Age: 38 BMI: 28.025","Age: 33 BMI: 38.9","Age: 20 BMI: 31.79","Age: 25 BMI: 41.325","Age: 46 BMI: 33.44","Age: 34 BMI: 34.21","Age: 19 BMI: 35.53","Age: 19 BMI: 30.495","Age: 18 BMI: 37.29","Age: 62 BMI: 36.86","Age: 25 BMI: 23.465","Age: 19 BMI: 20.7","Age: 27 BMI: 25.175","Age: 28 BMI: 26.98","Age: 59 BMI: 28.785","Age: 29 BMI: 26.03","Age: 24 BMI: 29.3","Age: 51 BMI: 39.7","Age: 48 BMI: 30.78","Age: 39 BMI: 26.22","Age: 48 BMI: 31.13","Age: 18 BMI: 33.66","Age: 19 BMI: 20.3","Age: 33 BMI: 18.5","Age: 53 BMI: 26.41","Age: 40 BMI: 41.69","Age: 24 BMI: 23.655","Age: 56 BMI: 33.725","Age: 47 BMI: 29.545","Age: 33 BMI: 32.9","Age: 62 BMI: 37.4","Age: 18 BMI: 40.28","Age: 45 BMI: 24.035","Age: 21 BMI: 28.975","Age: 58 BMI: 22.77","Age: 24 BMI: 33.63","Age: 31 BMI: 27.645","Age: 47 BMI: 32.3","Age: 64 BMI: 39.7","Age: 38 BMI: 19.475","Age: 61 BMI: 36.1","Age: 44 BMI: 36.48","Age: 45 BMI: 39.805","Age: 31 BMI: 29.26","Age: 22 BMI: 23.18","Age: 39 BMI: 31.92","Age: 64 BMI: 35.97","Age: 18 BMI: 30.14","Age: 46 BMI: 30.8","Age: 36 BMI: 34.43","Age: 60 BMI: 24.32","Age: 63 BMI: 33.1","Age: 35 BMI: 23.465","Age: 51 BMI: 34.2","Age: 25 BMI: 32.23","Age: 57 BMI: 28.1","Age: 55 BMI: 33.535","Age: 51 BMI: 25.4","Age: 40 BMI: 29.9","Age: 50 BMI: 36.2","Age: 33 BMI: 33.44","Age: 41 BMI: 28.8","Age: 23 BMI: 27.36","Age: 61 BMI: 44","Age: 26 BMI: 27.265","Age: 60 BMI: 35.1","Age: 44 BMI: 32.34","Age: 22 BMI: 28.31","Age: 44 BMI: 27.5","Age: 24 BMI: 33.99","Age: 34 BMI: 35.815","Age: 62 BMI: 39.16","Age: 26 BMI: 46.53","Age: 18 BMI: 26.18","Age: 19 BMI: 22.61","Age: 31 BMI: 39.49","Age: 24 BMI: 26.79","Age: 46 BMI: 24.795","Age: 52 BMI: 36.765","Age: 35 BMI: 27.1","Age: 39 BMI: 34.32","Age: 34 BMI: 23.56","Age: 22 BMI: 20.235","Age: 19 BMI: 40.5","Age: 50 BMI: 31.6","Age: 28 BMI: 29.26","Age: 21 BMI: 34.6","Age: 61 BMI: 38.38","Age: 54 BMI: 23","Age: 64 BMI: 26.41","Age: 36 BMI: 28.595","Age: 60 BMI: 18.335","Age: 32 BMI: 27.835","Age: 32 BMI: 31.5","Age: 57 BMI: 31.54","Age: 52 BMI: 47.74","Age: 23 BMI: 32.7","Age: 18 BMI: 31.35","Age: 49 BMI: 29.925","Age: 50 BMI: 26.22","Age: 26 BMI: 30","Age: 41 BMI: 32.6","Age: 52 BMI: 24.86","Age: 35 BMI: 35.815","Age: 36 BMI: 22.135","Age: 20 BMI: 30.59","Age: 32 BMI: 41.1","Age: 43 BMI: 34.58","Age: 32 BMI: 35.2","Age: 37 BMI: 34.105","Age: 19 BMI: 27.93","Age: 22 BMI: 32.11","Age: 59 BMI: 34.8","Age: 41 BMI: 23.94","Age: 56 BMI: 34.43","Age: 58 BMI: 30.305","Age: 58 BMI: 23.3","Age: 45 BMI: 27.83","Age: 18 BMI: 33.33","Age: 28 BMI: 24.3","Age: 55 BMI: 37.715","Age: 43 BMI: 29.9","Age: 35 BMI: 27.61","Age: 22 BMI: 30.4","Age: 18 BMI: 30.03","Age: 58 BMI: 25.175","Age: 20 BMI: 22","Age: 18 BMI: 26.125","Age: 18 BMI: 28.31","Age: 51 BMI: 30.03","Age: 57 BMI: 25.74","Age: 23 BMI: 33.4","Age: 18 BMI: 36.85"],"hoverinfo":["text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text"],"type":"scatter3d","mode":"markers","opacity":0.9,"name":"no","marker":{"color":"rgba(204,70,120,1)","size":[55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55],"sizemode":"area","symbol":"x","line":{"color":"rgba(204,70,120,1)"}},"textfont":{"color":"rgba(204,70,120,1)","size":55},"error_y":{"color":"rgba(204,70,120,1)","width":55},"error_x":{"color":"rgba(204,70,120,1)","width":55},"line":{"color":"rgba(204,70,120,1)","width":55},"frame":null},{"x":[31,45,20,19,42,63,56,47,53,50,44,36,63,64,61,40,24,21,39,57,54,35,48,31,18,42,50,53,27,37,47,44,34,27,30,18,20,43,22,47,48,23,18,23,43,52,52,24,18,62,32,30,62],"z":[718.439595977252,-2474.32791442279,163.862855918855,-562.445485375792,-616.081182383175,1041.55323564354,3143.71837891788,-1850.34353618616,-329.494262305605,193.887932563564,-1236.68064644067,-616.402794832837,1147.52147126736,807.364109688744,594.406661754227,-1232.82071366354,16605.0826519252,-1285.19746053166,29.6819771721894,2743.04078714739,19918.060429483,626.492913010443,423.586209927445,13979.0506891743,-3493.20571614619,11456.4007187653,-2421.89033677694,-242.84702924508,-479.029109853149,6797.69454657003,78.8573345539335,1461.587870512,-226.944800443358,-3189.22618434969,-3021.13937201994,-1018.16082160738,-649.879939689017,-1729.18011680482,11659.663655504,-1866.54167909923,-148.166383813448,-3022.11817953986,168.85072278915,-1824.65965524794,-2388.22020395241,17923.5920514336,1999.42107871245,-2993.49281159062,-3952.66077544965,2796.88985886247,-342.732420896598,-1510.46944299202,614.74905595959],"y":[36.3,22.895,28.025,28.3,24.64,27.74,33.63,25.41,22.61,32.3,20.235,22.6,26.98,33.88,35.86,32.775,28.5,25.7,28.3,42.13,47.41,34.105,40.565,38.095,33.535,28.31,27.6,20.9,28.5,47.6,38.94,30.2,27.835,20.045,22.99,21.565,30.685,24.7,52.58,36.19,25.85,31.4,27.36,28.49,25.27,34.485,41.8,29.83,21.66,30.875,28.12,23.655,26.695],"text":["Age: 31 BMI: 36.3","Age: 45 BMI: 22.895","Age: 20 BMI: 28.025","Age: 19 BMI: 28.3","Age: 42 BMI: 24.64","Age: 63 BMI: 27.74","Age: 56 BMI: 33.63","Age: 47 BMI: 25.41","Age: 53 BMI: 22.61","Age: 50 BMI: 32.3","Age: 44 BMI: 20.235","Age: 36 BMI: 22.6","Age: 63 BMI: 26.98","Age: 64 BMI: 33.88","Age: 61 BMI: 35.86","Age: 40 BMI: 32.775","Age: 24 BMI: 28.5","Age: 21 BMI: 25.7","Age: 39 BMI: 28.3","Age: 57 BMI: 42.13","Age: 54 BMI: 47.41","Age: 35 BMI: 34.105","Age: 48 BMI: 40.565","Age: 31 BMI: 38.095","Age: 18 BMI: 33.535","Age: 42 BMI: 28.31","Age: 50 BMI: 27.6","Age: 53 BMI: 20.9","Age: 27 BMI: 28.5","Age: 37 BMI: 47.6","Age: 47 BMI: 38.94","Age: 44 BMI: 30.2","Age: 34 BMI: 27.835","Age: 27 BMI: 20.045","Age: 30 BMI: 22.99","Age: 18 BMI: 21.565","Age: 20 BMI: 30.685","Age: 43 BMI: 24.7","Age: 22 BMI: 52.58","Age: 47 BMI: 36.19","Age: 48 BMI: 25.85","Age: 23 BMI: 31.4","Age: 18 BMI: 27.36","Age: 23 BMI: 28.49","Age: 43 BMI: 25.27","Age: 52 BMI: 34.485","Age: 52 BMI: 41.8","Age: 24 BMI: 29.83","Age: 18 BMI: 21.66","Age: 62 BMI: 30.875","Age: 32 BMI: 28.12","Age: 30 BMI: 23.655","Age: 62 BMI: 26.695"],"hoverinfo":["text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text","text"],"type":"scatter3d","mode":"markers","opacity":0.9,"name":"yes","marker":{"color":"rgba(13,8,135,1)","size":[55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55,55],"sizemode":"area","symbol":"circle","line":{"color":"rgba(13,8,135,1)"}},"textfont":{"color":"rgba(13,8,135,1)","size":55},"error_y":{"color":"rgba(13,8,135,1)","width":55},"error_x":{"color":"rgba(13,8,135,1)","width":55},"line":{"color":"rgba(13,8,135,1)","width":55},"frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>

---
## R setup

.pull-left[
<img src="data:image/png;base64,#img/tidyverse_hex.png" width="80%" style="display: block; margin: auto;" />
]
.pull-right[
<img src="data:image/png;base64,#img/tidymodels_hex.png" width="80%" style="display: block; margin: auto;" />
]

Amazing book: **Tidy modeling with R**

---
## Compressed tree visualization